Prevent parallel functions from forking Ark#1308
Conversation
thomasp85
left a comment
thomasp85
left a comment
There was a problem hiding this comment.
LGTM - I assume the suggestion from Kevin was not possible and we must error here?
DavisVaughan
left a comment
DavisVaughan
left a comment
There was a problem hiding this comment.
Seems nice to me too! Was just curious about how it worked so did a drive by review
|
@thomasp85 @kevinushey Kevin's suggestion could work in principle but to get it 100% reliable would be a lot of work. We could start by swapping all R frontend methods like read-console or write-console to no-ops. This would handle code that does But what about all the background threads, various R hooks that we might forgot, etc? This would require a large audit and comprehensive testing to get right. And this is just the beginning, because we've then introduced a permanent invariant that the process might be forked that we need to keep in mind as we keep developing Ark. The parallel computing landscape is now in a different place than 10 years ago, so it seems reasonable to fail with an error and recommend other solutions such as mirai/purrr. |
3 participants
Addresses posit-dev/positron#3817
This shims
mcfork(), the forking primitive used by all entry points in the parallel package. The shim fails with an informative R error.In addition,
makeForkCluster()is also shimmed because it swallowsmcfork()errors with an uninformative message.If
mc.coresis set to 1,mcfork()is not called as the {parallel} function falls back to serial computation, which is the same behaviour as on Windows where forking is not possible.Positron Release Notes
New Features
Bug Fixes
future::multisession(), mirai, orpurrr::in_parallel()(Problem with forked (multicore) code in ark positron#3817).